Repository navigation
fix(python-uv): preserve compiled bytecode after packaging - #926
Conversation
| # Use the exact interpreter found by UV instead of repeating a version request that | ||
| # could resolve differently when the install command runs. | ||
| if target_python: | ||
| args.extend(["--python", target_python]) |
There was a problem hiding this comment.
[GENERAL] The --python pin is no longer needed and makes an optional optimization change how the install itself runs.
_compile_bytecode() invokes the discovered interpreter directly:
command = [python_executable, "-m", "compileall", "-f", "-q", "--invalidation-mode", "unchecked-hash", target_dir]The resulting .pyc files depend only on that interpreter and the installed sources — not on which interpreter uv used. Meanwhile the installed sources are already governed by --python-version and --python-platform (both handlers always pass platform="linux" plus an architecture). So --python contributes nothing to the bytecode outcome, but it does override uv's own interpreter selection for the whole uv pip install invocation — including the interpreter used for any sdist build — replacing whatever uv would have chosen (e.g. an active virtualenv) with whatever uv python find happens to return.
Dropping the pin keeps the install behavior identical to today and confines the new code path to the compilation step:
if python_version:
args.extend(["--python-version", python_version])with target_python used only for _compile_bytecode().
| if rc == 0 and stdout: | ||
| return stdout.strip() | ||
| diagnostic = (stderr or "").strip() or (stdout or "").strip() or "no diagnostic output" | ||
| LOG.warning("Could not locate target Python %s via uv (exit code %d): %s", python_version, rc, diagnostic) |
There was a problem hiding this comment.
[GENERAL] A missing target interpreter is the normal case for cross-version builds, so LOG.warning here will surface a scary message on healthy builds.
LOG.warning("Could not locate target Python %s via uv (exit code %d): %s", python_version, rc, diagnostic)Building a python3.9 function on a host that only has 3.12 installed — or any build in an image whose interpreter differs from the runtime — hits this path, and --no-python-downloads guarantees uv will not fill the gap. Since bytecode compilation is a best-effort optimization whose absence has no effect on the produced artifact, this belongs at LOG.debug (or LOG.info), otherwise users see a warning in sam build output for a build that succeeded exactly as intended. Note this is amplified by the default in comment 1: with compilation on by default, most cross-version builds will emit it.
The same applies to the "Target Python version is unavailable" warning at line 155 — _extract_python_version() raises when there is no runtime, so a falsy python_version reaching install_requirements() only happens for direct library callers.
bnusunny
left a comment
There was a problem hiding this comment.
Thanks, unchecked-hash is the right approach. Two blockers:
Functional tests fail. 11 tests in tests/functional/workflows/python_uv/ fail with TypeError: ... expected str instance, bytes found, because FakeUv returns bytes for uv python find.
Please make bytecode opt-in. With compile_bytecode defaulting to True, every build ships .pyc files. That makes artifacts larger and builds slower, and the output varies with the build machine's interpreter. Could the default be False?
bnusunny
left a comment
There was a problem hiding this comment.
Both points from my last review are still open. Since 2026-09-16 the branch has only had merges from develop.
- Functional tests still fail: 11 of 13 in
tests/functional/workflows/python_uv/fail locally, and the same unit-functional jobs fail in CI on this head.FakeUv.run_uv_commandreturns bytes, sofind_pythonreturns bytes and building the compile command fails withexpected str instance, bytes found. Make the fake return str, at least forpython find. - Bytecode is still on by default (inline).
This also conflicts with #887 in install_requirements and tests/unit/workflows/python_uv/test_packager.py, so whichever PR merges second will need a rebase.
|
|
||
| # Add configuration arguments | ||
| target_python = None | ||
| if config.compile_bytecode: |
There was a problem hiding this comment.
UvConfig.compile_bytecode still defaults to True. On develop the flag is unused, so this change makes every build run compileall and ship .pyc files. Please change the default in UvConfig to False, update test_uv_config_defaults, and fix the DESIGN.md example.
…ests FakeUv now returns str, like SubprocessUv, so find_python no longer breaks the compileall command. compile_bytecode defaults to False. Unit tests and DESIGN.md are updated, and a functional test covers the opt-in unchecked-hash path.
bnusunny
left a comment
There was a problem hiding this comment.
Changes are in and CI is green. Thanks.
vicheey
left a comment
There was a problem hiding this comment.
Thanks for sticking with this through the review rounds. Unchecked-hash bytecode as an opt-in is a good outcome for #924/#925.
One small thing before merge: the PR description still says the default stays True and covers "the default enabled path". Could you update it to match the opt-in default, since it'll likely become the commit message?
# Conflicts: # aws_lambda_builders/workflows/python_uv/packager.py # tests/unit/workflows/python_uv/test_packager.py
uv rejects --no-compile-bytecode when UV_COMPILE_BYTECODE is set
("cannot be used with '--compile-bytecode'"), which failed every build
in that environment. Bytecode is compiled separately with unchecked-hash
invalidation, so remove the variable for the install command only.
bnusunny
left a comment
There was a problem hiding this comment.
Thanks for carrying this through, @xujiantop-crypto. Unchecked-hash bytecode compiled with the target interpreter is the right fix for #924, and opt-in is the right default for #925.
To save a round trip, I pushed two commits to your branch:
3b4ebd6merges develop. The conflicts were test-only (test_packager.py,test_utils.py) and I kept both sides.0f20821fixes a build failure:install_requirementsalways passes--no-compile-bytecode, and uv rejects that flag wheneverUV_COMPILE_BYTECODEis set in the environment (the argument '--no-compile-bytecode' cannot be used with '--compile-bytecode'). The install now drops that variable from its subprocess env (built fromoriginal_environ(), so the PyInstallerLD_LIBRARY_PATHhandling stays), and a new unit test covers it. Verified with real uv 0.13.9: withUV_COMPILE_BYTECODE=1set, the build failed before and succeeds after. The python_uv unit and functional suites pass (161), also with the variable set.
Remaining items, all non-blocking:
- The opt-in can't be reached yet. The workflow builds
UvConfigfrom theuv_*build options (workflow.py:153), andcompile_bytecodeisn't one of them, so it's alwaysFalsein real builds. That's fine as a first step: it turns #925's dead option into a real one that is off by default. Please say so in the description, and we'll track auv_compile_bytecodebuild option as a follow-up. - PR description: as @vicheey noted, it still says the default stays
True. It will likely become the squash commit message, so please update it to the opt-in default, and mention that--no-compile-bytecodeis now always passed andUV_COMPILE_BYTECODEis ignored.
|
|
||
| # Add configuration arguments | ||
| target_python = None | ||
| if config.compile_bytecode: |
There was a problem hiding this comment.
Nothing upstream sets compile_bytecode=True yet. UvConfig is built from the uv_* build options (workflow.py:153), and they don't include it, so this branch only runs in tests today. OK for this PR; worth a line in the description.
| def find_python(self, python_version: str) -> Optional[str]: | ||
| """Find an already-installed interpreter matching the requested Python version.""" | ||
| rc, stdout, stderr = self.run_uv_command( | ||
| ["python", "find", "--no-project", "--no-python-downloads", python_version] |
There was a problem hiding this comment.
uv python find <version> checks discovered virtualenvs before managed and system interpreters. That's fine, because the magic number depends only on the minor version and .pyc is arch-independent. If you want to rule out a non-CPython interpreter, cpython@<version> is an explicit request. Not required.
|
|
||
| def _compile_bytecode(self, python_executable: str, target_dir: str) -> None: | ||
| """Compile reusable bytecode with the interpreter selected for the target runtime.""" | ||
| command = [ |
There was a problem hiding this comment.
Consider -j 0 for compileall. Large dependency trees (#924 measured 977 files) compile serially otherwise.
|
I tested this branch ( As @bnusunny noted, though, |
|
Thanks for testing it. Agreed: I changed |
vicheey
left a comment
There was a problem hiding this comment.
Thanks for carrying this through. One thing is still open from the earlier rounds: the description says the default stays True and covers "the default enabled path", but the code defaults to False. Could you update it to say:
- the default is off, and the option can't be reached from a real build until #973
uv pip installalways gets--no-compile-bytecode, soUV_COMPILE_BYTECODEand a project's uv bytecode setting no longer apply to it
Optional: -I on the compileall call, and -- before the version in uv python find.
… unset The builder drops UV_COMPILE_BYTECODE from `uv pip install` because uv rejects it together with --no-compile-bytecode. Before aws#926 a truthy value made uv write timestamp .pyc; since then it silently produced none. Read the variable in the workflow when the build option isn't given and turn on unchecked-hash compilation with the target Python instead. Values are parsed like uv's boolish parser (1/true/t/yes/y/on vs 0/false/f/no/n/off, case-insensitive); anything else logs a warning and stays off. An explicit uv_compile_bytecode: false wins. Enabling via the variable is logged at INFO. Also document the bytecode behaviour in DESIGN.md, fix the stale "options fail on requirements.txt" note, and keep the existing tests hermetic against a developer's own UV_COMPILE_BYTECODE.
Issue #, if available: Fixes #924
Part of #925. The build option that turns this on is added in #973, and that PR is the one linked to close the issue.
Description of changes
UvConfig.compile_bytecodeeffective when it is set, and change its default toFalse: bytecode compilation is opt-in. Nothing in the workflow sets it yet; feat(python-uv): add uv_compile_bytecode build option #973 adds theuv_compile_bytecodebuild option for thatuv python find --no-project --no-python-downloads <version>--python, while retaining--python-versionfor dependency resolution--no-compile-bytecodetouv pip install, so uv and project uv configuration (uv.toml/[tool.uv]) never produce timestamp-based bytecode that SAM packaging would invalidateUV_COMPILE_BYTECODEfrom the environment of theuv pip installsubprocess. uv rejects--no-compile-bytecodewhen that variable is set (the argument '--no-compile-bytecode' cannot be used with '--compile-bytecode'), so it is ignored rather than allowed to fail the build. The environment still comes fromoriginal_environ(), so the PyInstallerLD_LIBRARY_PATHhandling is keptpython -m compileall -f -q --invalidation-mode unchecked-hash <target>so packaged.pycfiles remain valid after ZIP timestamps are normalizedUV_COMPILE_BYTECODEenvironment case in unit tests-fis intentional: it replaces any timestamp-based.pycalready present in an incremental target instead of allowingcompileallto skip it.Description of how you validated changes
On the current head (develop merged in):
tests/unit/workflows/python_uvandtests/functional/workflows/python_uv: 161 passed, also withUV_COMPILE_BYTECODE=1setUV_COMPILE_BYTECODE=1: the install failed with the flag conflict before the env fix, and succeeds after itruff check aws_lambda_builderspassedEarlier, on the original change:
sample.cpython-313.pyc.pycheader and confirmedflags=1(unchecked-hash), rather than timestamp invalidation.pycfrom develop +UV_COMPILE_BYTECODE=1are all stale afterwards, while unchecked-hash.pycfrom this change stay valid, with a cache tag matching the target runtime (3.9 / 3.11 / 3.12 / 3.13)git diff --checkpassedChecklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.